QTM 385 - Experimental Methods

Lecture 03 - Natural and Quasi Experiments

Danilo Freire

Emory University

Hello, everyone! 👋
Hope you’re all doing well! 😉

A good research question should produce knowledge that solves real-world problems and guides policy decisions, with a practical and credible research design
Theory is essential in research design, whether implicit or explicit, as it helps generate hypotheses, informs design choices, and guides inference strategies
Operationalisation is the process of translating theoretical concepts into measurable variables, such as turning “social isolation” into the frequency of social interactions
Pre-registration involves filing research designs and hypotheses publicly to reduce bias, improve credibility, and distinguish between pre-planned and exploratory analyses
The reproducibility crisis in science highlights the need for transparent and replicable research, with pre-registration and pre-analysis plans helping address this issue

The EGAP research design form provides a blueprint for creating robust research designs, covering key components like hypotheses, population, intervention, outcomes, and analysis
The MIDA framework (Model, Inquiry, Data, Answer) helps researchers simulate and diagnose their designs before implementation, ensuring they can answer their research questions effectively
DeclareDesign is a tool that allows researchers to declare, diagnose, and design their studies using the MIDA framework, improving the quality and credibility of research
A two-arm trial is a common experimental design where units are randomly assigned to treatment or control groups, and the average treatment effect (ATE) is estimated
Alignment between research design and theoretical frameworks is necessary for generating credible and actionable results, even when experiments are not feasible

The experimental ideal: why we should keep it in mind even when we use observational data
Natural experiments: what they are and how they can help us estimate causal effects
Quasi-experiments: how they differ from natural experiments and the challenges they pose
The quasi-experimental toolkit:
- Matching
- Instrumental variables
- Regression discontinuity
- Difference-in-differences
Internal and external validity: the trade-offs between them
Cumuative knowledge: how different research designs contribute to it

You already know by now that the most credible way to estimate causal effects is through randomised experiments
This is because experiments address confounding by randomly assigning units to treatment and control groups
- By confounding, we mean the presence of unobserved variables (\(Z\)) that affect both the treatment and the outcome
This means that:
- All covariates are balanced between treatment and control groups
- The treatment is independent of potential outcomes (more on that in the next lecture)
Today, we will investigate this ideal a little further, and see how we can get closer to it even when we don’t have experimental data
But first, let’s see an example